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SYSTEM AND METHOD FOR COMMUNICATION 
WITH ENHANCED OPTICAL POINTER 

CROSS-REFERENCE TO RELATED APPLICATIONS 

This application is a continuation-in-part of copending U.S. patent 
5 application Ser. No. 09/906,218 filed July 16, 2001, which is a continuation of U.S. 
patent application Ser. No. 09/347,761 filed July 6, 1999, now U.S. patent 
6,275,214, the disclosures of which are hereby incorporated by reference in their 
entirety. 

TECHNICAL FIELD 

O 10 The present invention relates to a system and method for remotely 

f 0 communicating information using an optical pointer. 

ill 

g BACKGROUND ART 

s 

O As computers are increasingly being used for graphical presentations 

ffi and/or demonstrations, larger display devices are used to provide viewing by 

W 15 audiences of varying size. Many presentations, such as slide shows and the like, 
ry require relatively simple control of the computer during the actual presentation. 

Commands which advance or reverse slides or initiate a display sequence require 
only a basic user interface or remote control to communicate with the computer. 
However, more sophisticated presentations or demonstrations, such as used for 
20 software user training or promotion, require a more sophisticated interface or remote 
control to effectively operate the computer. Conventional strategies require the 
presenter to either remain witfam close proximity of the computer to operate the 
keyboard and/or pointing device (mouse, touchpad, track ball, etc.) or have an 
assistant perform the required operations. 

25 The present inventor disclosed a system and method for controlling 

a windows-oriented computer system using a remote pointer in U.S. Patent 
6,275,214. This system is quite valuable for controlling and/or demonstrating one 
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or more computer applications in an auditorium-style presentation. However, this 
system does not directly address the use of a computer keyboard and/or other input 
devices during such a presentation. Furthermore, while the system and mediod 
disclosed in U.S. Patent 6,275,214 work well for a single optical pointer, they may 
5 fail to work properly when one or more optical pointers are used in a collaborative 
presentation, or in the case of someone attempting to disrupt or interfere with the 
presentation. As such, a need exists for a system and method for optically 
communicating more complex information that may also include identification 
information to authenticate a system user. 

10 DISCLOSURE OF INVENTION 



The present invention provides a system and method that enable 
{0 optical conomunication of information to a computer using a camera or other optical 



detector in communication with the computer for capturing and processing optical 

CS images to detect the information. The information may include keyboard characters 

H 

_ ' 15 commands, command sequences, voice data, identification data, or other data or 

^3 control information. The present invention provides for secure optical transmission 

(U 

fy of information using a pseudo-random modulation strategy which may be used to 

p identify or authenticate one or more optical devices or pointers enabling multiple 

fU simultaneous users aM preventing unauthorized use. In addition to computer 

20 presentation applications, systenos and methods in accordance with the present 

invention may be used for personnel identification and secure high-bandwidth Ime-of- 

site communications^ for example. 



In one embodiment of the present invention, a method for 
communicating information to a remotely located cony)uter includes encoding the 

25 information by modulating at least one feature of an optical pointer, transmitting the 
encoded information to a remote location using the optical pointer, detecting the 
modulation of the at least one feature of the optical pointer, decoding the information 
based on die detected plurality of features of die optical pointer, and generating a 
command for the remotely located computer based on the decoded mf ormation. In 

30 another embodiment, a remotely controlled computer presentation system includes 
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a computer, a screen associated with the computer for displaying output from the 
computer, and a camera in communication with the computer and positioned to 
capture a plurality of image frames including at least a portion of the screen 
associated with the computer. The computer includes instructions for processing the 
image frames to detect modulation of at least one feature of an optical cursor 
projected onto the screen from an external optical pointer and instructions for 
decoding the modulated feature to generate a corresponding input for the computer. 

Another embodiment of the present invention includes an optical 
device which transmits encoded information by modulating at least one feature of the 
optical device to a remotely located optical detector. The optical detector may 
directly or indirectly detect the modulated feature to decode the information. In one 
application, the optical device is carried or worn by a user with encoded user 
identification information directly or indirecdy transmitted to a detector that decodes 
the information to identify the user. For indirect detection, a camera may be used 
to capture video frames (of a screen, wall, or other surface, for example) that include 
the modulated optical device feature. The captured frames are then processed to 
decode the information and identify the user. For direct detection, the optical device 
may transmit directly to a camera or other optical detector. 

In another embodiment of the present invention, a method and device 
for generating input to a computer system using an optical device, such as a laser 
pointer are provided. An optical device is used to generate a pattern of movement 
or other change of properties or position to activate or launch a customizable user 
interface to provide subsequent input to one or more applications running on the 
associated computer. For exanqjle, specific movement patterns, dwelling on a 
particular location, or pressing a button to change properties of an optical pointing 
device may activate or launch a keyboard application that displays or projects a 
keyboard. The optical device may then be used to generate a series of keystrokes 
using the keyboard application and transmit or transfer the series of keystrokes to 
another application on the computer in response to an appropriate "send" command 
or in response to closing the keyboard application, for example. The user interface 
may also include various context-sensitive menus to select appropriate commands 
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including a left-click, right-click, or application specific commands, for example, and 
may be customized by the user or an application developer. Multiple user interfaces 
may be activated using the optical pointing device to generate corresponding 
commands if desired. The method and device for providing a customizable user 
interface application may be used Independently with a commercially available 
optical pointer, or in combination with a modulating optical pointer as disclosed 
herein. 

The present invention provides a number of advantages relative to 
prior art strategies. For example, the present invention is valuable for controlling 
and/or demonstrating one or more computer applications in an auditorium-style 
presentation. The present invention enables input of any kind of information, such 
as keyboard, voice, mouse, or other ii^ut device. The bandwidth of the information 
input via the system and method of the present invention is limited only by the 
selection of the pointer features and attributes, video frame capture rate of the 
camera, and the classic Nyquist sampling restrictions. 

The present invention may be used to provide a secure system where 
the remotely located computer responds only to one or more authorized pointers 
which exhibit the correct modulation pattern. Such a secure device has many 
applications beyond the computer control in a presentation system. For example, die 
present invention may be used for personnel identification, secure high-bandwidth 
line-of-site communications, and applications where two or more secure pointers are 
required for operation. As another example, the present invention may be used in 
military applications to **paint'' or kradiate a target using visible or invisible 
wavelengths for optical-guided vehicles or devices. Using a secure optical pointmg 
device according to the present invention makes it very difficult to "spoof* or 
confuse the target tracker usmg an unauthorized optical device. The potential 
niunber of unique pointer identifications in such a system is limited only by the 
pomter design and the encoding methods selected. For a computer presentation 
application, a secure pointer according to die present invention enables multiple 
simultaneous users with pointers which may project a seemingly identical spot or 
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cursor, but which may be distinguished by the computer to readily track each 
pointer's position and commands. 

The above advantages and other advantages, objects, and features of 
the present invention, will be readily apparent from the following detailed description 
of the best mode for carrying out the invention when taken in connection with the 
accompanying drawings. 

BRIEF DESCRIPTION OF DRAWINGS 

FIGURE 1 is a block diagram illustrating a representative computer 
presentation application for use with a system or method according to the present 
invention; 

FIGURE 2 is a diagram illustrating projection of calibration reticles 
or fiducials and derivation of coordinates for a cursor projected by a pointer 
according to one embodiment according to the present invention; 

FIGURE 3 is a diagram illustrating sections, areas, and attributes of 
an optical pointer or device according to one embodiment of the present invention; 

FIGURE 4 illustrates representative shapes for areas of an optical 
pointer or device according to one embodiment of the present invention; 

FIGURE 5 illustrates representative frequencies or wavelengths for 
areas of an optical pointer or device according to one embodiment of the present 
invention; 

FIGURE 6 illustrates representative intensities for areas or sections 
of an optical pointer or device according to one embodiment of the present invention; 
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FIGURE 7 is a diagram illustrating the relationship between emitters 
and areas for a representative section of an optical pointer or device according to one 
embodiment of the present invention; 

FIGURE 8 is a flow chart illustrating operation of a system or method 
5 for transmitting information to a remote computer using an optical pointer or device 
according to one embodiment of the present invention; 

FIGURE 9 is a diagram illustrating an optical device used for personal 
identification and/or communication according to one embodiment of the present 
invention; 

FIGURE 10 is a diagram illustratiug a customizable user interface 
used to enter keyboard information or other information into another application 
running on a computer associated with a computer presentation system according to 
the present invention; and 

FIGURE II is a diagram illustrating a customizable user interface 
used to enter other commands into an application running on a computer according 
to the present invention. 

BEST MODE FOR CARRYING OUT THE INVENTION 

As used herein, the terms "optical**, "light" and related terminology 
are used in their broadest sense. As understood by those of ordinary skill in the art, 
20 such terms include a range of frequencies or wavelengths of the electromagnetic 
spectrum which encompass visible light, infrared, ultraviolet, and other such 
frequencies or wavelengths, as distinguished from the radio-frequency spectrum 
used in other forms of "wireless** control. 

Referring now to Figure I, a block diagram illustrating a 
25 representative computer presentation application with information communicated by 
an optical pointer according to one embodiment of the present invention is shown. 
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As used herein, the remote control of the computer is intended to include control of 
various application programs and/or the operating system of the computer, which 
preferably are controlled via a graphical user interface. However, as explained in 
greater detail below, the present uivention is also capable of transmitting keyboard 
characters and voice information (directly or through a customizable user interface) 
that may be used to remotely control either a text-based application or operating 
system, or to provide information and commands to a graphical user interfiace of an 
application program or operating system. In this embodiment, computer 10 is 
connected to a video projector 12 and a video camera 14. In this embodiment, video 
projector 12 projects an image 16 of the computer output onto a projection surface 
or screen, mdicated generally by reference numeral 18. Preferably, screen 18 is a 
"passive- screen, i.e. a substantially planar surface remotely located from computer 
10 and of a light color to provide sufficient contrast to the image of computer output 
generated by projector 12. Various objects may be used to provide such a surface, 
including projection screens, walls, or the like. Output from computer 10 may also 
be displayed on an "active" screen which would include a traditional computer 
screen 20 of laptop computer 10, or any other display device such as a monitor, 
television, plasma display, or similar "active" device. Such "active" screens may 
be used alone, or in combination with one or more "passive" screens without 
departing from the spirit or scope of the present mvention. 

As also illustrated in Figure 1, an optical cursor or graphic, indicated 
generally by reference numeral 22, is generated by a device other than computer 10 
and superimposed on im^e 16 which is output from computer 10, Cursor or graphic 
22 could be generated by any of a number of devices including another computer, 
projector, or the like. In the embodunent illustrated m Figure 1, cursor 22 is 
generated by a hand-held optical pointer 24 which preferably has the capability of 
modulating at least one feature or property of cursor 22. Preferably, the feature or 
property is modulated in a predetermined manner or fashion known by computer 10 
such that pointer 24 can be uniquely identified. As illustrated and described in 
greater detail with reference to Figures 3 and 4, optical pointer 24 may vary the 
color, shape (or number of shapes), size, intensity, illumination pattern, motion, 
and/or position of cursor 22 to generate one or more commands to remotely control 
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computer 10 (directly or via another user interface) according to the present 
invention. Optical pointer 24 may also communicate with additional input devices 
such as a keyboard or keypad 26 and headset 28 which preferably includes at least 
a microphone 30 and may include a speaker or earphone 32. Optical pointer 24 may 
communicate with keypad 26 and headset 28 via wired or wireless connection(s) 
depending upon the particular application. Keypad or keyboard 26 is preferably a 
one-handed keyboard, such as those available from Infogrip, Inc. and can be worn 
by the user and operated with the opposite hand. Of course, optical pointer 24 may 
also be integrated into various other input devices, such as keypad 26, for example, 
and worn by the user, if desired. 

Preferably, camera 14, or other detector, repeatedly captures an image 
including at least a substantial portion of image 16 generated by projector 12. In an 
alternative embodiment where an active screen is utilized, camera 14 preferably 
captures at least a substantial portion of the active screen, i.e. computer monitor, 
display, or television. Computer 10 processes the captured image frames to 
determine at least one property of external cursor 22. In one embodiment of the 
present invention, computer 10 processes image 16 to determine at least the position 
of cursor 22 and generates an appropriate command or commands to move a 
computer generated cursor 34 to approximately the same position as cursor 24. The 
tracking of the optical cursor by the computer cursor may have a user or system 
specified offset deprading upon the particular application. Likewise, to provide 
sufficient stability to the tracking to accommodate user or system mduced jitter or 
other •*noise*' the tracking preferably includes a filtering fonction and necessarily 
lags position of the optical cursor to some degree that depends on various system 
parameters. As explamed in greater detail below, computer 10 may also process 
captured frames of image 16 to detect a predetermined or known modulation of 
various other features or properties of external cursor 22 to uniquely identify optical 
pointer 24 or identify optical pointer 24 as one of a plurality of authorized pointers 
that may be used to control computer 10. In one embodiment, a pseudo-random 
modiilation strategy is used such that only those pointers following the same pseudo- 
random pattern synchronized with the computer will be recognized. Position or 
context-dependent commands may emulate a •*left-click" or "right-click" command 

8 



KLH 0105 PUS 



generated by a traditional computer pointing device, such as a mouse, track ball, 
touch pad, or the like. In addition to allowing simultaneous use of multiple pointers, 
or use of only an authorized pointer, modulation of one or more feamres of optical 
pointer 24 may be used to transmit a variety of information to computer 10. For 
5 example, keypad information from keyboard 26 or voice information from 
microphone 30 may be transmitted via modulation of various features of optical 
pointer 24 as explained in greater detail with reference to Figures 3 and 4. 

In one embodiment of the present invention, computer 10 may 
periodically generate reticles or fiducials, indicated generally by reference numeral 
10 36, to calibrate or register the image captured by camera 14 relative to image 16 on 
screen 18. Preferably, each of the four comers of the image of the computer output 
□ contains a reticle or fiducial and the fiducials are used to delineate the "active" 

i% trackmg region where cursor 22 is detected. The fiducials may be any appropriate 

in shape such as a right-angle or cross as illustrated. For applications where screen 18 

15 is not substantially planar, fiducials 36 should cover more of the screen. For 
^'""^ example, fiducial lines may extend horizontally or vertically across screen 18. 

Preferably, projector 12 (when used), camera 14, and screen 18 are 
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Co stationary and remain substantially fixed to facilitate the calibration (registration) and 

tracking process. However, the present invention could also be used in applications 
20 where one or more of these devices changes position, although detection of the 
properties of external cursor 22 becomes more complex and computationally 
intensive. The calibration or registration process may be repeated automatically at 
predetermined intervals, based on a user request, and/or when cursor 22 is not 
detected. In one embodiment of the present invention, reticles or fiducials 36 are 
25 progressively moved toward the position of cursor 22 to simplify processing of the 
image frames to detect cursor 22. In this embodiment, only the area delineated by 
fiducials 36 is searched or scanned to detect cursor 22. If cursor 22 is not located 
within the area defined by fiducials 36, then: position is either reset to. or 
progressively expanded toward, the original comers of the processed hnage until 
30 cursor 22 is detected. 
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Figure 2 represents a single image frame 40 captured by video camera 
14, Preferably, frames are repeatedly captured and processed to detect modulation 
of at least one property or feature of cursor 22. Typical frame capture rates are 
thirty or sixty frames per second. Preferably, the frame capture rate of the video 
5 camera and/or the output of an active screen or projector are selected to minimize 
aliasing and other interference within the frequency bands of interest. Such 
interference may be produced by the beat frequency or difference frequency between 
the refresh rate of the screen or projector and the video camera. Any such 
interference effects may also be reduced by appropriate filtering of the captured 
10 image. Filtering may be performed as part of the image processing by the computer, 
or may be performed by appropriate hardware as part of the image capture process. 

□ Captured image frame 40 preferably includes a substantial portion of 

p the computer output image, represented generally by reference numeral 42. As 

in illustrated in Figure 2, computer output image 42 may occupy a convex quadrilateral 

j| 15 area within captured frame 40. Image 42 will not be rectangular if either or bodi of 

M the projector and the camera are not located on the line which passes perpendicularly 

s 

p through the center of the screen. Preferably, the computer embeds or supenmposes 

the reticles or fiducials 36 within image 42. Processing of captured image 40 may 
utilize line and intersection pattern recognition depending upon the particular type 
S 20 of fiducials utilized to locate comers Cl\ C2\ C3\ and C4'. Determination of the 
locations of CI', C2\ C3\ and C4' and the properties of the external cursor is 
simplified because their identifying characteristics are known a priori. Identification 
of items within a static image and identification of dynamic motion of features may 
be accomplished using any number of known image processing techniques, such as 
25 those described in "MACHINE VISION" by Jain, Kasturi, and Schunk, published 
by McGraw-Hill, ISBN 0-07-032018-7. The present invention is independent of the 
particular image processing techniques utilized to identify or detect the properties of 
the external cursor used to remotely control the computer. An exemplary method for 
determining position or location information of the external cursor is provided 
30 below. 
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The locations of corners Cl\ C2\ C3\ C4\ and cursor 22 may be 
determined for a classic video quadrant system having horizontal coordinates 
represented by X and vertical coordinates represented by Y, with Y=0 representing 
the topmost scan-line. The number of scan-lines determines the maximum Y value. 
The positions corresponding to the coordinates for X=0 are the first (left-most) pixel 
in each scan-line, and the number of pixels per scan-line determines the maximum 
X value. The line segment CI— C2 corresponds to the X-axis, and segment CI— C4 
corresponds with the Y-axis. Points interior to image 42 are represented with 
normalized (T, U) coordinates where CV has coordinates (T = 0.0, U = 0.0), and 
C3' has coordinates (T = 1.0, U = 1.0). The normalized coordinates can then be 
scaled to an appropriate resolution for image 40. Coordinates T and U can be 
determined for any point in a given video frame using a pair of parametric equations. 
First, X*Y coordinate pairs for each of the comers CV through C4' are represented 
as: 

Cr = (Xr, Yl') 
C2' = (X2', Y2') 
C3' =- (X3', Y3') 
C4' = (X4', Y4') 

The parametric equation for a line is P(Q) = PC - Q(PO - PI), where 
Q is replaced by T and then U in this example. The T-axis is defmed using 
cr— C2' and C4'— C3' while the U-axis is defined using Cr— C4' and C2'— C3\ 
Because the embedded quadrilateral defining image 42 is not guaranteed to have 
parallel boundaries, the mapping from (X,Y) to (T,U) is not a simple rectangular 
mapping in the general case. However, the mapping may be derived as described 
below. 

The parametric equations for the embedded quadrilateral boundaries 
defining image 42 are given by : 

Upper boundary (LTl): p(T) = CI' - T(Cr - C2') 
Lower boundary (LT2): p(T) = C4' - T(C4* - C3') 
Left boundary (LUl): p(U) - CI' - U(Cr - C4') 



11 



KLH 0105 PUS 



Right boundary (LU2): p(U) = CT - U(C2' - C3') 

The mapping in (T,U) may then be defined by selecting either the LT pair or the LU 
pair and defining a parametric equation in the other variable (either U or T, 
respectively) as follows: 

1: p(T) - LUl - T(LU1 - LU2) 

or, equivalently: 

2: p(U) = LTl - U(LT1 - LT2) . 

Equation 1 represents how the U-axis scales and/or rotates as it traverses the (T,U) 
space from its location at T = 0, to its location at T = 1 . Equation 2 represents how 
the T axis scales and/or rotates as it traverses the (T,U) space from U = 0, to U = 
1. As indicated above, either equation may be used to determine a parameteric 
equation in flie other variable. In this example. Equation 1 is selected and the 
equations for LUl and LU2 are substimted into Equation 1 to generate Equation 3 
as follows: 

3: p(T,U)=[Cr-U(Cr-C4')] - T{[Cl'-U(Cr-C4')] - (C2'-U(C2'-C3')]} 

To shnplify the notation, it is useful to define a shorthand notation representing the 
differences or deltas between coordinates of the comer points as follows: 

di:cr-cr 

d2: CT - C3' 
d3: C4' - C3' 
d4:Cr^C4' 

Where a difference value applies to the difference between values of only one 
coordmate, i.e. only the X or only the Y coordinate values, the difference is 
represented by either an X or Y, respectively, between the "d" and the number. For 
example, dX2 represents the quantity (X2' - X3*), while dY4 represents the quantity 
(Yr - Y4'). The (X\Y') point pairs are then substituted for CI', C2\ C3\ and C4' 
in Equation 3. with the delta notation used to simplify the resulting equations. 
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Depending upon how the four points multiplied by U are paired, one of the following 
equations results: 



4: p(T,U) = [Cr - U * d41 - T[dl - U(dl - d3)I 

or 



5: p(T,U) = [Cr - U * d41 - T[dl - U(d4 - d2)] 



Equations 4 and 5 are equally valid and each represents a pair of linearly independent 
equations, one in X and the other in Y because p(T,U) is represented by a generic 
coordinate pair (X, Y). Equation 5 is selected for this example and split into the X 
and Y equations: 



6: X = Xr - U * dX4 - T * dXl 4- UT(dX4 - dX2) 
7: Y - Yr - U * dY4 - T * dYl + UT{dY4 - dY2) 



Equations 6 and 7 are then solved for T and U, respectively, to produce equations 
8 and 9 as follows: 

" dXl - U(dK4 - dX2) 

9' U ^ 17^ - y - r * dYl 
' ' dY4 - IidY4 - dY2) 

Because Equations. 8 and 9 are linearly independent* either equation can be 
substituted into the other. In this example. Equation 9 is substimted into Equation 
8 to obtain: 



f/ V ^^yi^ - Y ' T * dYl 



XI' - X - dX4^ 
J. ^ dY4 - ndY4 - dY2) 

' dXl-idX4^ dX2)^^^- r- dYl 

dY4 - ndY4 - dY2) 

Which is now only an equation in T for an arbitrary point havmg coordinates (X,Y) 
in the original space. Solving Equation 10 for T yields a simple quadratic in T: 
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11: 0 - iir^ + Br + c 

where 

A = dYl(dX4 - dX2) - dXl(dY4 - dY2 ) 
B = dXl*dY4 - (Yl^Y)(dX4-dX2) - dYl*dX4 + (Xl-X)(dY4-dY2) 
C = dX4*(Yl-Y) - dY4(Xl-X) 

Equation 1 1 may then be solved using the quadratic formula. In Equation 11 , A is 
constant, while B and C vary depending upon the values of X and Y. As such, if the 
lines formed by extending the segments Cr— C2' and C4*--C3' intersect, there will 
be two solutions to Equation 11, one of which is the T coordinate of the point of 
intersection, the other of which is the correct value for T. If the lines do not 
intersect, the single solution will be the correct value for T. Once the correct value 
of T is determined, it is back-substituted into Eqn. 9 to determine the corresponding 
value for U. 

The above example demonstrates that once Cr, C2\ C3\ and C4' are 
known, and the (X, Y) coordinates of cursor 22 are determined, a straight-forward 
computation yields values for (T,U) which represent the **mouse coordinates" for 
computer generated cursor 26 of the computer output. Because the T-U coordinate 
system has been normalized, any coordinates with T or U outside of the normalized 
range (0 to 1) can be ignored for tracking purposes because they do not fall within 
unage 42 of computer output. To determine the scaled values for the coordinates of 
computer generated cursor 26, T is multiplied by the horizontal resolution and U is 
multiplied by the vertical resolution of the captured image 40, Once the calibration 
or registration process has determined CI*. C2', C3\ and C4\ modulation and or 
movement of at least one property or feature of cursor 22 is monitored or tracked by 
repeatedly capturing and analyzing frames as illustrated and described with reference 
to Figure 5. In one embodiment of the present invention, position of cursor 22 is 
determined only while cursor 22 is within projected image 42 while various other 
modulated properties of cursor 22, such as shape, color, size, etc. are monitored as 
long as cursor 22 is within captured image 40, i.e. even when cursor 22 is outside 
of projected image 42. 
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While rectangular/perspective distortions, such as pincushion, can be 
taken into consideration and normalized as described above, correction/mapping of 
various other types of distortion may also be performed in accordance with the 
present invention. Depending upon the particular application, multiple mappings 
may be used for various regions of the image, rather than using a single mapping for 
the entire captured image. Likewise, the particular source of the distortion does not 
necessarily impact the correction for that distortion. For example, the above 
approach may be modified to account for distortion introduced by lens systems of the 
camera (or other optical detector) and projector apart from the distortion diat can be 
introduced by projection onto a non-planar surface or projection at an angle. All of 
diese distortions can be compensated or corrected using appropriate calibration 
screens, similar to but perhaps more complex than the reticles described and 
illustrated in the present application. As the levels of distortion increase, more and 
more complex calibration screens may be needed. Depending upon the particular 
type of distortion, the mathematics may be significantly more complex than the 
example above. However, as processor speed and capability continue to improve, 
the computational complexity has less of an impact on system performance. 

Referring now to Figure 3, an output array of a representative optical 
pointer according to one embodiment of the present invention is shown. Optical 
pointer 50 includes a plurality of sections (S1-S4) indicated generally by reference 
numerals 52, 54, 56, and 58, Each section includes a predefined group of one or 
more areas delineated in the Figure by dashed lines and represmted generally by area 
60. As illustrated in Figure 3 for section represented by numeral 56, the areas 
forming a section need not be contiguous or symmetric. In this example, areas ( 1 ,4) , 
(1,5), (1,6), (2,6), (3,6), (3,3) (4,3), (5,3), (6,3), (6,4), (6,5), and (6,6) form a 
smgle section S4. As illustrated and described in greater detail with reference to 
Figure 7, each area 60 may be generated by a portion of an emitter, a dedicated 
emitter, or a cluster of emitters acting together. Preferably, each area includes a set 
of attributes, the modulation of which can be used to distinguish a given area among 
successive image frames. For most implementations, all areas will have identical 
attribute sets. This is not required although simplifies the design and 
implementation. 
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Figure 4 illustrates various values for a representative shape attribute 
of an optical pointer area according to one embodiment of the present invention. As 
illustrated, shapes 70-88 take various forms that may include a rectangle 70, square 
74, circle 76, or line, such as lines 78, 80, 84, and 86. As illustrated by the lines, 
a particular geometric shape may have various orientations. Likewise, a particular 
shape may be composed of smaller sub-shapes as represented by double diamonds 
72. Similarly, cross 88 and "X** 82 may be composed from discrete intersecting 
lines or segments. 

Figure 5 illustrates various values for another representative attribute 
of an optical pointer area according to one embodiment of the present invention. As 
illustrated schenaatically in Figure 5, a "color** attribute may have various values. 
As described above, the color attribute actually represents a wavelength or range of 
wavelengths (or frequencies) for each value which may correspond to visible colors 
generally illustrated as red 90, yellow 92, orange 94, blue 96, and green 98. In 
addition, infra-red 100 and ultra-violet 102 wavelengths may be used, among others, 
depending upon the particular implementation. Of course, the camera or other 
receiver or detector should be capable of detecting and distinguishing between the 
various color attribute values for proper operation of the system. 

Figure 6 illustrates another representative attribute for optical pointer 
areas according to one embodiment of the present invention. As schematically 
depicted in Figure 6, an intensity attribute may have various values, generally 
represented by low 100, medium 112, and high 114 intensities, respectively. Again, 
the number of intensities and degree of variation will depend upon the particular 
application. However, the camera or other detector in combination with the 
processing software should be able to accurately and reliably discriminate between 
the values for each of the area attributes for error-free operation. Of course, to 
optimize performance of the system, any of a number of error detection and 
correction algorithms may be employed in combination with appropriate encoding 
strategies. 
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Referring now to Figure 7, a diagram illustrating the relationship 
between emitters and areas of a representative optical pointer according to one 
embodiment of the present invention is shown. Figure 7 represents a section 52 (Fig. 
3) which includes a total of six areas 60. As also illustrated, emitters 120, 122, 124, 
5 and 126 are capable of generating one or more areas 60, or a portion of an are when 
clustered with other emitters. Stated differently, an area is generated by a portion 
of an emitter, a dedicated emitter, or a cluster of emitters acting together. For 
example, emitter represented by reference numeral 120 generates a total of four 
areas 60: A(i u; A^^j; A^.i>; and A^,2)- contrast, area A(, 3^ is generated by emitters 
10 122, 124 acting together and area A^^^ is generated by a dedicated emitter 126. 

u As illustrated and described with reference to Figures 3-7, each 

area may have one or more attributes with each attribute having one or more 

£0 defined values. For an attribute of shape, these values may include square, 

iR 

triangle, circle, or line. For an attribute of color, these values may include red, 
^5 15 blue, green, white, or IR. For an attribute of intensity, values may include low, 
a medium, and high. Note that "off" is not an intensity attribute, but rather an 

|H absence of any attribute for the given area. 

The number of values for a given attribute is defined as 
lU VA(Attribute). In the above definition for attribute values, VA(shape) is 4, 

20 VA(color) is 5, and VA{intensity) is 3. 

The set size for a given area's attribute set is defined as VS(Area), 
and is the product of the different VAQ values availatjle for the given area, plus 
one for the "ofT state. In the above definition for attribute values, and assuming 
that all areas in an optical pointer are designed identically, VS(Area) is 4 * 5 * 3 
25 4- 1 , or 61, i.e. the product of VA(x) where x is each of the attributes, plus one 
for the "empty set" when the area is "off". Of course, if all areas within a pointer 
are "off', the pointer cannot be located. If, however, the position is being 
tracked, or at least one area is not "off", the locations may be inferred. 
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A pointer has a "dominant shape" which is determined by the 
perimeter, circumference, or "hull" around all of its emitters/areas when all are 
active at some predetermined intensity. A pointer can have embedded shapes or 
"sub-shapes" which are formed by various combinations of emitters/areas turned 
off while others are active. All of the sub-shapes will fit within the outline 
formed by the dominant shape, but may be entirely different. For example, the 
dominant shape of a pointer may be square, but the various areas within the 
square may be capable of generating stars, circles, triangles, etc, in the same 
manner that a dot matrix is rectangular but is capable of generating a multimde of 
shapes, all smaller than the dot matrix displaying them. 

Each section of the optical pointer has a set of "states", formed by 
the areas and area attributes within that section. The size of this set is the product 
of the number of areas forming the section, and VS(Area) if each area is identical 
in design. If the Areas are differing in design, the size of this set is the sum of 
VS(Area) for all areas within the section. This size is defined as MaxSn, where 
"n" represents the section nimiber if the pointer has more than one section. As an 
example, if a given section had 10 identical Areas with VS(Area) = 61 as in the 
example above, MaxS for that Section would be 61 * 10, or 610. 

A Section is "modulated" when one or more of its areas changes its 
attribute set. To uniquely identify a pointer, one or more sections may be 
modulated with a predetermined or pseudo-random pauem which is known by the 
detector/processor, i.e. the attribute set changes in a predetermined manner to 
transmit a pointer "signature** which can be identified by the detector. The 
modulation rate for a section is given by the number of times per second that a the 
attributeset of a section changes. This is defined as ModSn. The bandwidth of a 
section is defined as the product of MaxS and ModS. As an example, if ModS is 
30 for the example section with MaxS of 610 above, the bandwidth for that 
section would be 18,300. In the most simple encoding scheme, this is also 
equivalent to the bit-rate, but there are several encoding schemes which yield 
multiple bits per system-state-change, which could be used to give higher data 
throughput. Increasing the number of areas per section will also yield higher 
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rates, e.g. a section consisting of a square 5x5 array of these same areas would 
have a bandwidth = 30 * 25 * 61 = 45750, which is sufficient for CD-quality 
audio (40kbits/second) with a sinq)Ie encoding strategy or scheme. 

The mtensity, color, and shape modulations described herein are 
achievable using a variety of known techniques. As an illustrative example, the 
digital light processing (DLP) technology available from Texas Instruments, Inc. 
One or more DLP arrays could be used to modulate the color/intensity/shape of 
an individual emitter, or could be used to create multiple "emitters" from a single 
emitter, each controllable by sections of a given DLP array. For purposes of the 
present invention, a hand-held DLP projector driven by a dedicated single-board 
computer could provided the necessary functionality of the optical pointer with an 
associated keyboard or keypad and/or microphone/speaker. The on-board 
computer senses the buttons pressed by the user or sanq)les the sound to generate 
a digital data stream, then generates the image which is subsequently projected by 
the hand-held DLP projector. Such a pointer could use a rectangular dominant 
shape with a single white-light emitter, with a rectangular array of areas (the 
pixels in the DLP element), each of which has a fixed square shape, but which 
can change color and intensity. The number of areas is the total number of pixels 
in the DLP element and the number of sections is one. 

In one possible implementation of a "complete control" system, the 
user has a keyboard (preferably a one-handed one) and wears a microphone. The 
embedded computer driving the optical pointer output encodes the keystrokes and . 
speech into the light projected by the optical pointer. The video frames are 
captured by the camera and processed to extract the encoded information and 
decode the keystrokes/audio, and pass them on to a graphical user interface 
application program or operating system. This permits data entry, as well as 
voice command via the pointer, without limitations of RF transmission distance, 
quite literally enabling a full computer demonstration on a screen at one end of a 
football field, for example, by a presenter up in the stands at the other end of the 
field. The presenter is not tied to the computer by wires or RF transmission 
limitations. If the presenter can see his pointer on the screen, so can the 
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captxiring camera or other detector, since it is preferably located in close 
proximity to the projector generating the output image. 

Another possible implementation of a "complete control'' system 
activates a customizable user interface to generate and store a series of keystrokes 
and/or commands that are subsequently provided to another program or operating 
system. The customizable user interface may be activated using a conventional 
optical pointing device based on a button press, motion, dwell, pattern of 
movement, or the like. Once the user interface is activated, the optical pointer 
can be used to generate commands or keystrokes. A "send" command, 
corresponding to a button on the interface or associated with modulation of 
another pointer property, for example, transfers the series of keystrokes or 
commands to one or more background applications or the operating system. In 
one embodiment, as illustrated in Figiure 10, the customizable user interface 300 
resembles a traditional keyboard. User interface 300 is activated when a 
predetermined command is transmitted using a laser pointing device. In this 
example, the optical pointer cursor dwells on a particular location 302 (with some 
specified tolerance) for a predetermined time to activate user interface 300. Any 
or all of the predetermined parameters, such as the type of command to activate 
the user interface, the time associated with a dwell, etc. may be user-specified or 
selected depending upon the particular application. Once activated, user interface 
300 may include various regions 304 corresponduig to alphanumeric keys 306 
and/or command keys 308. Buttons or keys on user interface 300 may be 
operated in a similar fashion as the commands used to activate user interface 300. 
For example, a key press may be indicated by dwelling on a corresponding region 
304 of user interface 300 for a predetermined time. Alternatively, a key press 
may be indicated using a button on the optical pointer to change one or more 
properties of the optical cursor, such as the shape, color, intensity, modulation, 
etc. In one embodiment, a key press is indicated by turning the optical pointer 
"ofT then "on". The characters may be stored and displayed in a buffer 312 for 
modification by the user. After the desired series of one or more commands or 
keystrokes are entered, another button or command 310 is used to transfer the 
input to one or more application programs or the operating system. Although 
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illustrated using an exemplary keyboard, one of ordinary skill in the art will 
appreciate that the customizable user interface according to the present invention 
may be used to provide any number of user or developer specified keypads, 
menus, or the like to generate input for application programs or the operating 
system. For example, in the embodiment illustrated in Figure 11, a command 
menu 320 is displayed when activated by the optical pointer. Conmnand menu 
320 mcludes buttons corresponding to mouse commands, such as a left-click 322. 
right-click 324, ciick-and-drag 326, and double-click 328. , Additional control 
features or commands may also be activated using other buttons, such as cancel 
330 and a command or button to pop-up a keyboard 332 as illustrated and 
described with reference to Figure 10. 

In a sufl[iciently complex poixUer, different sections could be 
devoted to encoding different "channels" of information. For example, one 
section could carry mouse events, anodier voice information, a thkd keyboard 
strokes, a fourth GPS coordinates of the pointer, etc., limited only by design and 
video capture/processing rate limits. The kinds of data which can be transferred 
through this pointer are limited only by the imagmation. 

The motions introduced into the pointer are typically the result of 
hand motions of the presenter, but could also be generated by attaching the 
pomter to other body areas, or mounting the emitter array on a variety of 
platforms capable of moving in one or more durections, such as a small actuator 
driven X-Y slide, or a rotating table, or some other device capable of givmg the 
desired motion. In this manner, a pattern of motion may be used as a possible 
mechanism for generating commands/data for the computer system. This is easily 
accomplished by tracking pointer location across tune and analyzing the resultmg 
"motion history" for patterns with defined significance. A further extension to 
this is to track patterns across time and define a grammar to translate the resulting 
information into data and/or conunands. This is readily accomplished by defining 
tokens for each pattern, then using tools such as LEX and YACC (from UNIX) to 
translate the pattern stream into actions, code sequences, and/or data. This may 
also be extended into patterns of patterns, etc., nested as deeply as desired. 
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Unintended motions of the optical pointer may be introduced when 
the user clicks a button on the optical pointer to change one or more properties. 
The greater the distance between the optical pointer and the object acting as the 
projection screen, the greater the effect of a small movement because clicking a 
button generally results in an angular deflection rather than a linear displacement. 
By physically or mechanically isolating any button presses from the device that 
generates or controls the position of the optical cursor, the cursor position is 
easier to control and has improved stability. In one embodiment, a separate 
command modxile includes at least one button and communicates button presses to 
the optical device, which then generates an appropriate command as described 
elsewhere herein. The command module may communicate with the optical 
cursor device using wired or wireless commtmication. For example a low power 
RF signal may be used because of the small distance between the devices. This 
type of commimication link would be more reliable than using a similar RF link to 
couple to a remotely located computer that may be several hundred feet away. 

An alternative method for providing mechanical isolation of 
movement associated with key presses on an optical pointer is to provide a smal 
gyroscopic stabilizer or angular momentum wheel within the optical pointing 
device. A stabilizer resists unintended angular deflections while allowing a user 
to apply intentional translational force to depress a button or to change position of 
the optical cursor, for example. In effect, the stabilizer would provide a damping 
force to reduce or minimize unintended movements of the optical cursor. 

Either of the above mechanistns used alone will significantly 
improve the ability of a given user to place the optical ciu-sor on a desired location 
with accuracy and generate a command associated with a button press. Used 
together, these mechanisms should provide substantial improvement because they 
each affect different aspects of the inaccuracy or instability. Physical isolation of 
one or more buttons addresses the case of a user intentionally applying a force to 
depress a button and unintentionally producing angular deflection that changes 
optical cursor position. The mechanical isolation in the form of a stabilizer or 
damping mechanism, such as a momentum wheel or gyroscope, addresses the case 
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where the user has minor hand tremors or other inadvertent hand motions. Both 
mechanisms » whether used alone or in combination, permit intentional angular 
deflection or translational movement. The use of a momenmm wheel or 
gyroscope may require an additional power pack (e.g. a belt-worn battery pack) 
although advances in battery technology may allow a self-contained pointer and 
momentum wheel with internal battery power. 

If pointer orientation is tracked (through designated orientation 
indicator areas and/or properly designed dominant shape), additional "movement" 
attributes may be communicated, such as rotation, size, focus, etc., as data 
generation mechanisms. For example, consider a pointer having a dominant 
shape corresponding to a simple "angle", like the " > " symbol on a keyboard. 
This shape has an obvious orientation. If the user of the pointer rotates the 
pointer by twistmg their wrist or by some other mechanism, the orientation of the 
pointer changes and can be used to generate a command (such as a "mouse rotate 
clockwise") to control the system. This is but one advantage of the present 
invention over traditional input mechanisms for windows-oriented computer 
systems, such as a mouse, touch-pad, or tablet, which are not designed to track 
orientation of the input device. 

Because the effective magnification of the image projected by the 
optical pointer changes as a function of the distance between die pointer and the 
screen, when the pointer is closer to the screen/detector, the overall size will be 
smaller than when it is farther away. The relative change in size depends on a 
ratio of the distance the pointer moves relative to the maximum distance between 
the pointer and the screen. Depending upon the overall distance, this 
characteristic may not be useful in an auditorium-style presentation. However, in 
a small environment it can be used to generate additional commands or sequences 
of commands after appropriate calibration. For example, because the size of the 
dominant shape shrinks as the pointer approaches the screen/detector, this can be 
translated into a "mouse push" command, enabling 3-D control (X, Y, and 
Depth), or a more intuitive zoom control. As the pointer is pulled away from the 
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screen/detector, the change in size can be translated into a *'mouse pull", the 
converse of the "mouse push" . 

Further 3-D style commands could be generated by interpreting the 
geometric distortion of the dominant shape and/or embedded sub-shapes due to 
5 die angle at which the pointer is projected onto the screen/detector. At near 
perpendicular angles, very little distortion occurs, other than the size 
increase/decrease due to distance. At oblique angles, substantial distortion 
occurs. The distortion is readily mapped back to rectangular using a similar 
approach to that described above., and an approximate angle/direction to the 
10 pointer can be determined. This could be used to generate 3-D rotational 
commands or sequences of commands, for example, in a CAD environment. 

Referring now to Figure 8, a flowchart illustrating operation of one 
embodiment of a system and method according to the present invention is shown. 
5^ As will be appreciated by one of ordinary skill in die art, the flowchart illustrated 

5 15 in Figiu-e 8 may represent any of a number of known processing strategies such as 

m 



m 

m 



event-driven, interrupt-driven, multi-tasking, multi-threading, and the like. 



Similarly, the present invention is independent of the particular programming 
p language, operating system, or computer platform which may be used to 

implement the invention. As such, various stc^s or functions illustrated may be 
20 performed in the sequence illustrated, in parallel or in some cases omitted 
altogether. Likewise, the order of processing illustrated is not necessarily 
required to achieve the objects, features,, and advantages of the invention, but is 
provided for ease of illustration and description. 



Preferably, the logic of Figure 8 is implemented primarily in 
25 software which is executed by a microprocessor-based computer, such as a laptop 
or desktop computer, represented generally by computer 10 of Figure 1. Of 
course, various portions or functions of the illustrated logic may be implemented 
in software, hardware, or a combination of software and hardware. Likewise, 
various steps or functions may be performed by a user or device controilii^ an 
30 optical pointer. The flowchart of Figure 8 illustrates one **loop*' and its 
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operations are preferably repeated at periodic time intervals or as required by 
some internal or extemsd event, as known by those of skill in the art. Preferably, 
control logic represented in Figure 8 is embodied in data stored on a computer 
readable storage medaun, such as a floppy disk, hard disk, CD-ROM, tape, or 
other magnetic, optical or combination storage media. The computer readable 
storage medium includes data representing instructions executable by a computer 
to detect information generated by an optical pointer through modulation of at 
least one feature or attribute. 

Block 150 of Figure 8 represents encoding information for 
transmission by an optical device, such, as an optical pointer, according to one 
embodiment of the present invention. As described above, the information may 
include keypad or keyboard information or characters, voice information, text 
messages^ identification Information or the like. The information may be encoded 
using any of a number of known encoding strategies. Preferably, a built in 
processor or computer of the optical pointer encodes the information and produces 
corresponding signals to modulate at least one feature of the pointer as 
represented by block 152. As described above, preferably one or more sections 
of the optical pointer are modulated by changing corresponding attribute sets. 
The optical pointer transmits or projects the encoded information onto a 
destmation or remote location as represented by block 154. In one enabodunent, 
the destination is a remotely located screen or surface. However, depending upon 
the particular application and implementation, the remote location may be the 
camera or detector, i.e. the optical pointer transmits directly to an optical 
detector, such as a CCD array of a camera or the like, rather than indirectly via a 
surface imaged by a camera. 

The optical device feature modulation is detected as represented by 
block 156. In one embodiment, a camera is used to capture a series of image 
frames which include at least a portion of the remote destmation or screen. The 
image frames are then processed, preferably by an associated computer, to detect 
and decode the encoded information as represented by block 158. Corresponding 
data and/or commands are then generated for the remote computer based on the 
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decoded infonnation as represented by block 160. This information may also be 
used to identify or authorize the pointer and/or user as represented by block 162 
with corresponding commands generated only if the pointer is authorized, or 
associated with a particular one of a plurality of pointers being used 
simultaneously. When used in a personal identification application, the 
conunands generated may be used to control access to a particular building,' room, 
area, or computer, for example. 

Figure 9 illustrates a representative personal identification and/or 
communication application of an optical device according to one embodiment of 
the present invention. Users 200, 202 may use optical devices 204, 206 for 
identification and/or optical line of sight communication. For example, optical 
device 204 may include a transmitter or emitter array portion 210 and an optional 
receiver or detector portion 212. User 202 may use an optional headset 216, 
keyboard (not shown) or other input device to provide information to optical 
device 206 for encoding and transmitting as described above. Modulation of 
feamres of hand-held optical device 206 is detected directly by receiver/detector 
212 of optical device 204. The received signal is then decoded and provided to 
user 200 via headset 220 or other output device depending upon the particular 
application. As illustrated m Figure 9, optical device 204 may be worn by user 
200, if desired. 

As also illustrated in Figure 9, optical device 204 may be used to 
uniquely identify user 200 by periodically (or when commanded by the user) 
transmitting an optical signal 230 containing identification information. Optical 
signal 230 may be directly delected by a corresponding receiver/detector 232 
which processes the signal to decode the information based on the detected feature 
modulation and identify user 200. The decoded information may be used to 
generate commands to gain access to a particular room, building, area, or 
computer, for example. Alternatively, optical device 204 may transmit encoded 
identification information via optical signal 230 to a siuface 236 for indirect 
detection by a camera 238, which repeatedly captures, stores, and processes video 
frames as described above. 
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As such* the present invention provides a number of advantages 
relative to prior art strategies and is valuable for controlling and/or demonstrating 
one or more computer applications in an auditorimn-style presentation. As 
described above, the present invention enables input of any kind of information, 
such as keyboard, voice, mouse, or other input device. The bandwidth of the 
information input via a system and method of the present invention is limited only 
by the selection of the pointer features and attributes, video frame capture rate of 
the camera or detector, and the classic Nyquist sampling restrictions. 

While embodiments of the invention have been illustrated and 
described, it is not mtended that these embodiments illustrate and describe all 
possible forms of the invention. Rather, the words used in the specification are 
words of description rather than limitation, and it is understood that various 
changes may be made without departing from the spirit and scope of the 
invention. 
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